Journal of Astronomy & Earth Sciences Education - June 2017 


Volume 4 , Number 1 


Development Of The EGGS Exam 
Of GeoloGy Standards To Measure 
Students’ Understanding Of 
Common Geology Concepts 

Sarah Katie Guffey, University of Wyoming, USA 
Timothy F. Slater, University of Wyoming, USA 
Stephanie J. Slater, CAPER Center for Astronomy & Physics Education Research, USA 


ABSTRACT 

Geoscience education researchers have considerable need for criterion-referenced, easy-to-administer, easy-to- 
score, conceptual surveys for undergraduates taking introductory science survey courses in order for faculty to 
monitor the learning impacts of innovative teaching. In response, this study establishes the reliability and validity of 
a 28-item, multiple-choice, pre- and post- EGGS Exam of GeoloGy Standards. EGGS addresses 11 concepts derived 
from a systematic analysis of the overlapping ideas from national science education reforms: NGSS, AAAS 
Benchmarks, Earth Science Literacy Principles, and NRC National Science Education Standards. Leveraging best- 
practices for creating and field-testing items, EGGS emphasizes natural student language over technical scientific 
vocabulary, leverages illustrations over students ’ reading ability, specifically targets students ’ misconceptions, and 
covers the range of topics most geology educators expect general education students to know. EGGS is argued to be 
valid and reliable with college introductory science survey students based on standard measures, including clinical 
interviews with students and expert review. 
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"n a time of international concern and debate about a changing global environment, geoscience educators 
are broadly interested in improving the teaching and learning of introductory geology survey courses for 
non-science majors and future teachers at the undergraduate level. Based on our analysis of textbook sales, 
nearly 800,000 undergraduates enroll in an introductory science survey course covering the Earth and space sciences 
each year. Reflecting the rapidly widening diversity of students now attending institutions of higher education, nearly 
half of students who take these courses do so at 2-year community colleges or tribal colleges. Yet, few of the faculty 
teaching these courses have had substantial formal training in pedagogy or contemporary teaching practices as part of 
their graduate coursework. For many of their students, a Geology 101 course often marks the end of many college 
students’ formal education in science. As such, Geology 101 courses serve as a unique—and perhaps final—forum to 
highlight the intimate relationships between science, technology, and society for students, many of which will go on 
to become teachers. Lawrenz (2005) and colleagues report that as many as 40% of students in introductory science 
survey courses eventually become certified teachers who will serve a critically important role for systemic change in 
the enterprise of science education. 


At the same time more and more educators want to know how much their students learn when using competing 
teaching strategies. There is considerable interest in finding widely accepted assessment instruments that can be easily 
used across teaching contexts so that comparisons can be made readily (Freeman, Eddy, McDonough, Smith, 
Okoroafor, Jordt, & Wenderoth, 2014). Like other scientific disciplines, the geoscience education community has long 
been calling for a comprehensive conceptual survey that effectively and efficiently covers the broad range of topics 
taught in most introductory geology survey courses. Although several competing geoscience education instruments 
do exist (viz., Cervato, Rudd, & Wang, 2007; Iverson, Steer, &Manduca, 2012; Libarkin, 2008; Libarkin & Anderson, 
Copyright by author(s); CC-BY 25 The Clute Institute 





Journal of Astronomy & Earth Sciences Education - June 2017 


Volume 4 , Number 1 

2005a), the broader impacts of these efforts have seen limited adoption. This is due in large part to a general lack of 
broad community consensus about which geology topics should be taught, and which topics should be tested. In other 
words, the geoscience education community has had trouble finding a pathway to meet a commitment to the mantra, 
“teach what you test, and test what you teach” (Guffey, Slater, Schleigh, Slater, & Heyer, 2016). If one could ascertain 
a wide consensus of what geology concepts needed to be taught, then one might be able to create a widely adopted 
geology assessment instrument. 

Much of our state-of-the-art understanding about college-level science teaching and learning has been summarized in 
the National Research Council’s publication, Discipline-Based Education Research: Understanding and Improving 
Learning in Undergraduate Science and Engineering (DBER, 2012). The overarching challenge to the higher 
education community is that learning and retention in college and university science courses could be enhanced by 
faculty using empirically validated teaching practices and materials that engage students in active learning (DBER, 
2012). The critical need resulting from this NRC committee report is for the community to develop an easy-to- 
implement, classroom-ready assessment to help college faculty systematically evaluate the impact of various teaching 
innovations. 

Other discipline-based education research fields seemingly have been able to reach a reasonable level of consensus 
about which topics should be taught. Consider the recent example from the Earth sciences domain of astronomy 
(Singer, Nielsen, & Schweingruber, 2012; Slater, Slater, Heyer, & Bailey, 2015). A roadmap astronomy education 
research agenda paper by Adams and Slater (2000) drove a decade’s worth of development of single-topic conceptual 
surveys (e.g., Bailey (2012) and colleagues’ Star Properties Concept Inventory SPCI, among many others cited by 
Bailey and Lombardi (2015)), before the broader community decided that what was most needed was a single, 
comprehensive, easy-to-use, easy-to-score, assessment instrument based in natural student language, as opposed to 
the vocabulary of scientific-jargon. In response, the Test Of Astronomy STandards , hereafter referred to as TOAST\ 
was developed by Stephanie Slater (2014) and colleagues. Committing to the test design specification of being 
comprehensive, but still short enough to be manageable, the team isolated the absolute minimum list of concepts a 
successful student was expected to know and understand by analyzing national reform documents. The geoscience 
education community could adopt the process used to develop the TOAST in order to successfully create an easy-to- 
use, easy-to-score assessment instrument for geology. It is to meet these needs that this article is specifically addressed. 

BACKGROUND AND CONTEXT 

Several conceptual and knowledge diagnostic inventories exist within the geoscience education community. The most 
recent effort has been a project widely known as the Geoscience Literacy Exam , GLE. There is only limited work on 
this project published in the scholarly literature as yet, although there is some work exists in the non-refereed grey 
literature (Geraghty-Ward, Iverson, Steer, & Manduca, 2015; Gosselin, Manduca, Bralower, & Mogk, 2013; Iverson, 
Steer, & Manduca, 2012; Olson, 2014; Steer, Iverson, & Manduca, 2013; and Steer, 2012). In brief, the GLE consists 
of 52 multiple choice and 30 essay questions based upon big ideas and grand challenges of Earth systems science 
identified through the NSF-funded InTeGrate project. Because of the relative newness of this project it is difficult to 
predict the extent to which the GLE will find widespread adoption across the geoscience education community. 
However, one naturally wonders about the manageability of such a large bank of questions by the typical college 
professor. 

The most well-funded effort to create widely-used assessment instruments to measure student’s understanding, 
including Earth science, is that of th q Mis conceptions-Oriented, Standards-based, Assessment Resources for Teachers, 
MOSART (2015) project. In terms of Earth science, the MOSART project team carefully created 60 multiple-choice 
questions aligned with the NRC National Science Education Standards (1996), in three grade-level bands, and 
collected substantial field-test data. With the clear exception of Sadler and colleagues’ (2009) widely cited article 
about the importance of developing an instrument to assess students’ knowledge of astronomy and space science, 
MOSART assessment instruments related to Earth science have made surprisingly little penetration into the scholarly 
literature. MOSART’s lack of widespread adoption and use by faculty is likely due in part to the current difficulty in 
obtaining the test items as researchers and faculty must go through an online tutorial before being granted access. 
Additionally, part of the lack of widespread adoption might also be due to too many items existing in the item bank 
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to be practical for most faculty to use in a pre-test and post-test format. Moreover, a cursory view of the instrument’s 
face validity suggests that successfully answering the items requires students to have a relatively high English reading 
ability as the instrument contains few illustrations and, as a result, suffers from a high chance of inadvertently testing 
reading comprehension instead of the desired level of Earth sciences conceptual understanding, which is consistent 
with the written cautionary warnings provided by the developers. In the end, for our purposes, the most compelling 
reason not to use the MOSART items was because it was aligned with only one of the existing science education 
reform efforts, the NRC’s National Science Education Standards (1996) and, as such, did not represent a broad 
community consensus of all the existing efforts at establishing a coherent set of learning targets for geology. 

The largest community effort to date is the Geoscience Concept Inventory, GCI (Libarkin, Anderson, Beilfuss, & 
Boone, 2005b). The overall goal of the GCI was to design an assessment instrument that would be a valid tool for use 
for all entry-level college students nation-wide, and which could be applied to a wide range of courses covering a 
variety of topics relevant to the Earth Sciences. However, there are several validity issues concerning the GCI. 

The first issue limiting the widespread adoption of the GCI pertains to criterion and content validity of the GCI. 
According to Field (2013), criterion validity determines if a test reflects a certain set of abilities relevant to the GCI’s 
concepts. Field (2013) defines content validity as “the extent to which a measurement reflects the specific intended 
domain of the content.” Libarken (2005a, 2005b, & 2006) and her colleagues used the largely insufficient approach 
of grounded theory to determine which geology concepts to include in the GCI. 

In the development of the GCI, thousands of hours of student interviews were performed and students also completed 
questionnaires related to geology concepts. After students’ responses and questionnaires were transcribed and coded, 
students’ responses were used to decide which content to include when generating the test items for the GCI, which 
is an ineffective method to develop test items. According to Johnson and Christensen (2014) in education research 
there are two accepted ways of determining the range and domain of content for an objective test of this nature, norm- 
referenced tests (NRT), which compares students to each other, and criterion-referenced tests (CRT), which compare 
students to a predetermined standard. In K-12 education, teachers are often interested in knowing if their students have 
learned the standards that were developed by content experts and curriculum developers. An example of a standards 
document is the Next Generation Science Standards (NGSS). An example of a validated CRT is the widely used 
Slater’s Test of Astronomy STandards (TOAST) (2014). All items on the TOAST are aligned with at least three national 
standards documents (NRC National Science Education Standards, AAAS Benchmarks for Science Literacy, and 
Achieve Inc’s Next Generation Science Standards), which addresses criterion validity. Once the items on the TOAST 
were aligned to standards, experts in the astronomy community reviewed the assessment to ensure content validity. 
The use of grounded theory for developing the GCI to determine which geology to include turned out to largely be 
unsuccessful in addressing content and criterion validity. 

The second weakness of the GCI pertains to construct validity, which is usually defined as the extent to which the test 
may be said to measure a coherent idea, in this case, the disciplinary structure and core concepts of geoscience (Huck, 
2013). When analyzing students’ responses, it is difficult for researchers to compare assessment results to what is 
already published about students’ conceptual understanding in geology. As suggested earlier, the GCI used grounded 
theory to identify student misconceptions; however, the resulting problem is that researchers are not be able to compare 
results from the GCI to previously validated literature in geoscience cognition. A connective, broad review of literature 
in geology cognition seems to be missing from the development of the GCI, adding to the observed lack of adoption 
by the community. 

The third issue interfering with the GCI’s widespread adoption by the community relates to face validity, which is a 
measure of how representative the assessment is at face value (Huck, 2013). The biggest concern about the GCI are 
the actual test items and the answer choices. Again, the use of grounded theory to develop test items and answers 
choices is suspect as a strategy. A few items on the GCI did not include the scientifically accurate answer choice as 
an option because responses from student interviews were used to create the answer choices. Some questions were 
later removed after IRT item response theory (e.g., Rasch modeling) analysis was performed on pre- and post-test 
results (Libarkin, Anderson, Deeds, & Callen, 2006). Results from the GCI showed that many items were considered 
stable; however, the test items that were removed due to the Rasch analysis were items that did not have a clearly 
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correct answer choice. This was inappropriately attributed to biased “gender and racial discrimination” on those 
specific questions because of the demographic characteristics of those who answered in particular ways (Libarkin & 
Anderson, 2005a). 

The last challenge faced by the general construction of the GCI is one of face validity. Overall, there are 73 questions 
on the GCI, which are too many questions according to researchers in assessment development (Boud & Falchikov, 
2007). When an assessment has too many questions, the researcher often encounters the issue of students developing 
test exhaustion and the researcher becomes unsure if students are marking incorrect answers because they don’t know 
the information or if they answer questions incorrectly because they are tired from taking the test. Generally to avoid 
test exhaustion, a good assessment will include 25 to 35 questions (Downing & Haladyna, 2006). 

In regard to the construction of the test items, it is known from expert assessment developers Miller and Linn (2000) 
that an item should only contain one correct or clearly best answer. There are several questions on the GCI that state, 
“Select all correct answers that apply.” According to Miller and Linn (2000), when an assessment asks students to 
“select all correct answers” this creates two problems. First, these types of questions are nothing more than a collection 
of true/false items presented in multiple-choice format. Best practice is that students should be able to answer a 
question without any given choices; otherwise, such items require a mental response of true or false to each alternative 
one at a time rather than a comparison and selection of alternatives. The second problem deals with the complexity of 
interpreting results from the answers of the “select all that apply” types of questions. The researcher is often 
questioning if students left the selection blank because they thought it was false and how would the researcher be able 
to determine if that was the case. This devalues these types of assessment questions. Another issue with item 
construction involves linking items on the test. For example, GCI item number 10 should not ask students about their 
response to GCI item number 2. Research in assessment development shows that asking students to complete 
additional tasks besides just answering the question, interferes with how the student responds to the question (Downing 
& Haladyna, 2006). 

Since 2008, there has been very little published about developing a new GCI instrument. The most notable is an un¬ 
refereed newsletter article published by Libarkin and colleagues (2011), which describes the need to revise the first 
Geoscience Concept Inventory (GCI v.1.0), stating, “We are now retiring GCI v.1.0 and rebuilding the GCI as a more 
community-based, comprehensive, and effective instrument.” These authors recognize the weaknesses in the GCI 
v.1.0, many of which were previously listed above. These authors also discuss the benefits of collaboratively revising 
the GCI v.1.0. Their article called for geologists, science educators, and instrument developers to participate as 
reviewers and authors of new questions. Since that time (2011), there has been no formal publication of the “new” 
GCI. Five years have passed and the geology education community is still using the original GCI. Such a situation 
suggests the community is ready for a new assessment instrument. 

It is reasonable to question the planned revisions for the new GCI and the weaknesses of those proposed revisions. 
Libarkin (2011) suggested the “expansion to more complex, wider ranging questions” (p. 1) which “will allow 
replicable assessment in advanced courses and across geoscience programs” (p. 1). Help was requested from “experts 
knowledgeable about issues students have understanding complex ideas (p. 2)” to “write, review, and test new 
questions” (p. 2). While this is one approach to developing a survey instrument, the researchers still faced two main 
challenges. 

First, the researcher was faced with the challenge of identifying the content to be included on the new instrument. It 
does not appear that any next generation GCI is intended to be aligned purposefully to any national standards 
documents—including the AAAS Benchmarks (1993), NSF Earth Science Literacy Principles (2008), NRC National 
Science Education Standards (1996), Achieve Inc.’s Next Generation Science Standards (2013b)—meaning even 
after proposed revisions the GCI will not be a true criterion-referenced test (CRT). The content included will be content 
that various individuals deem important. In education literature we’ve known from as early as the 1970s, that 
instruction and assessment design are directly linked to clearly stated learning objectives (Tamir & Jungwirth, 1972). 
The GCI v.1.0 wasn’t aligned to national science standards reform efforts and as a result lacks the wisdom of the 
broader community. 
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Libarkin (2011) recognized the weaknesses in the GCI v.10 and planned to revise it; however, as previously stated, it 
has been five years and the geology education community is still lacking a comprehensive assessment instrument 
designed to measure students’ general geology content knowledge. In the current project described here, our 
development effort leverages the strengths and mitigates the weaknesses of the GCI in hopes that science educators 
will find a new instrument to be fruitful and easy-to-adopt across the nation. 

Approach to EGGS Survey Development 

To avoid difficulties of earlier geoscience conceptual assessment projects, there were multiple stages used to address 
criterion and content validity of the proposed Exam of GeoloGy Standards , EGGS. The systematic development and 
validity process of the Test Of Astronomy STandards, TOAST (Slater, 2014) was closely followed during the 
development of the EGGS. The first stage included the development of a standard aligned consensus document 
(Guffey, Slater, & Slater, 2017, in review). This ensures that EGGS is a criterion-referenced test (CRT). The standards 
were aligned from the following four validated national science education reform standards documents: the American 
Association for the Advancement of Science (AAAS) Benchmarks for Scientific Literacy (1993), the National 
Research Council (NRC) National Science Education Standards (1996), the National Science Foundation (NSF) Earth 
Science Literacy Principles (2010), and the National Research Council (NRC), the National Science Teachers 
Association (NSTA), the American Association for the Advancement of Science (AAAS), and Achieve Inc’s, Next 
Generation Science Standards (2013b). The resulting consensus was distributed to 137 geology experts (with a 29% 
return rate) across the country requesting feedback and comments regarding the geology standards and was revised 
based on the received feedback. The alignment of standards on the consensus document was used as the base criteria 
on the EGGS. This systemic process resulted in the bare minimum of what students should know when leaving an 
introductory geology course. 

There were two purposeful efforts to address construct validity. First, as the goals consensus was finalized, but before 
test item construction began, a literature review of geology cognition was completed (Guffey & Slater, 2017, in 
review). Being aware of misconceptions and how people think about geology concepts helps facilitate the test item 
construction process. Questions were pulled from previous peer-reviewed, empirical literature in the geosciences. For 
example, Marques and Thompson (1997a) performed a study to identify middle and high school students’ 
misconceptions of plate tectonics. During the study students were asked particular questions to probe their thought 
process of plate tectonics. These are the types of questions that were used from peer-reviewed, empirical literature for 
the EGGS. The second step that addressed construct validity included the actual item construction, which was modeled 
after Slater’s TOAST (2014). All questions were written at a 7 th grade reading level to ensure that EGGS would not be 
primarily assessing reading ability, and predominantly assessing geology conceptual understanding. 

After identifying the questions used on the EGGS, the same process that was used to address criterion and content 
validity were used to start the process of assessing face validity. Geology experts reviewed the test items, provided 
feedback, and revisions were made. The last step to address face validity involved the distribution of the EGGS to 
individuals outside of geology and geology education. The outsider’s perspective helps to determine the extent to 
which EGGS is well designed and how it appears “at face value.” 

Under-girding all of this is the systematic identification of the most commonly agreed upon teaching goals for 
introductory geology teachers (Guffey, Slater, Slater, 2017, in review) and simultaneously a thorough investigation of 
the scholarly literature reporting students’ misconceptions related to these consensus goals is presented in detail 
elsewhere (Guffey & Slater, 2017, in review). Taken together, this motivates and enables our research and 
development effort to establish the validity and reliability of a new, easy-to-administer conceptual diagnostic for 
geology professors presented here. 


METHODS 

In an effort to establish the validity and reliability of an instrument for measuring students’ understanding of geology 
concepts, this effort employs a well-established sequential explanatory design (Creswell & Clark, 2011). In this 
approach, the underlying assumption is that the researcher knows the important variables and has access to quantitative 
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instruments for measuring the constructs of primary interest in addition to having the ability to return to participants 
for a second round of qualitative data collection. Ultimately, the researcher develops new questions based on 
qualitative results that cannot be answered with quantitative data, which leads to qualitative data collection and 
analysis. This study began with the quantitative strand, collecting closed-end data with a survey, the EGGS Exam of 
GeoloGy Standards , which is found in Appendix B. The researcher analyzed the quantitative results which were used 
to complete the following: 1) Refine the qualitative and mixed methods questions; 2) Determine which participants 
will be selected for the qualitative sample and; 3) Design qualitative data collection protocols. The researcher then 
collected open-ended data with protocols in the form of interviews. Qualitative data were analyzed using procedures 
of theme development to answer the qualitative and mixed methods research questions. Quantitative and qualitative 
results were interpreted separately. Mixing will occur when discussing to what extent and in what ways the qualitative 
results help explain the quantitative results. 

Participants 

The target population was non-science majoring undergraduate students, enrolled in introductory science survey 
courses in U.S. universities and colleges. The students were varied in terms of their GPA, major, number of science 
courses taken, and the degree they are pursuing. Criteria for selecting the participants included: 1) being enrolled in 
an introductory science survey course; 2) time period of Spring 2017; 3) must be in the beginning of the semester; and 
4) undergraduate student status. A total of 194 students completed the EGGS assessment with six of those students 
being selected for follow-up interviews. In the first phase, assigning unique numeric passwords to each participant 
allowed them to access the web-based survey, which protected the anonymity of the participants. In the second phase, 
the participants selected for interviews were assigned fictitious names, thus keeping the responses confidential. In 
addition, all the names and gender related pronouns were removed from the quotations used for illustrations. 

Quantitative Phase 

Data Collection 

For the first, quantitative phase, the EGGS Exam of GeoloGy Standards was administered online and accessed through 
SurveyMonkey. Active email addresses of potential participants were obtained through introductory science course 
professors and identified through other sources. The participants were recruited via e-mail a week before the beginning 
of the study. The data collection took place the first week of the Spring 2017 academic semester. The final version of 
the multiple-choice instrument used for this portion of the study is included in the appendix. 

Data Analysis 

Varma (2008) recommends conducting a detailed item-by-item analysis of multiple-choice items created for surveying 
conceptual understanding. Student responses were analyzed using the statistical analysis software program, IBM SPSS 
Statistics , to calculate Cronbach’s alpha values, inspect item difficulty levels, and item discrimination indices. To 
ensure internal consistency of the EGGS, reliability was established by calculating Cronbach’s alpha. To further 
establish the sufficient sensitivity of the instrument, two aspects of classic test theory were analyzed for the EGGS: 
item difficulty and item discrimination. Based on extensive and successful work in the realm of conceptual knowledge 
survey development cited earlier, this approach was judged to be a more fruitful approach than more often heralded, 
but rarely used correctly, contemporary IRT item response theory (Wallace & Bailey, 2010). 

The researcher randomly selected six students from those who volunteered to complete the EGGS and participate in 
think-aloud interviews. Using a straightforward clinical interviewing technique, interviewees were asked to elaborate 
(discuss their thinking out loud) on each question. The researcher only recorded field notes, which were compared to 
the geology misconceptions literature to see if students hold the same misconceptions found in the literature, or if new 
notions became apparent. 

Qualitative data were analyzed by performing a comparative analysis between the field notes collected in the interview 
and the geology misconceptions found in the literature. Upon completion of the analysis, the researcher developed an 
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organized list of themes, which included the geology misconceptions. The list of organized themes were compared to 
the quantitative results. At this point, the researcher synthesized geology misconceptions that weren’t identified upon 
completion of the quantitative analysis of the EGGS. 


RESULTS 

After several earlier iterations and pilot-testing of EGGS items, the final published 1.0 version of EGGS was 
administered to students enrolled in their first week of a college-level introductory science survey course designed for 
non-majoring undergraduates. Overall, 172 participants scored an average of 42% correct with a standard deviation 
of .014. The Cronbach Alpha was 0.69 and the average item discrimination was 0.34. Taken together with supporting 
interview data, we argue that this published version 1.0 of EGGS is a valid and reliable instrument to elicit students’ 
misconceptions. 

EGGS Concept 1: Earth’s crust is broken into plates, which slowly move in relationship to each other , driven by 
convection currents in the mantle . 


Figure 1. Item Analysis for EGGS Question 1 


Ql. About how far do Earth’s continents—and the plates they ride on—move in a single year? 

Response Labels 

Item difficulty 

Item discrimination 

a. A few inches 

0.81 

0.45 

b. A few hundred feet 

0.11 


c. A few miles 

0.05 


d. Continents and plates do not really move 

0.03 



_ Figure 2. Item Analysis for EGGS Question 5 _ 

Q5. Which letter on the diagram above illustrates a location where Earth’s plates are moving APART? 

Cross-section illustrating Earth's crustal plate boundaries 
showing oceanic crust and continental crust (not to scale) 
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_ Figure 3. Item Analysis for EGGS Question 6 _ 

Q6. Scientists often talk about Earth’s tectonic plates and their role in mountain formation, volcanoes, and earthquakes. Which 
of the following figures most closely represents the location of Earth’s tectonic plates? 




Response Labels 

Item difficulty 

Item discrimination 

a. 

0.51 

0.63 

b. 

0.36 


c. 

0.11 


d. 

0.02 



Three questions were designed or adapted to measure students’ understanding of plate tectonics as the central theory 
of geology. Question 1 was adapted from AAAS Project 2061 Science Assessment (AAAS, n.d.) assessment item 
PTO19001 in order to measure student’s understanding of plate movement rates. AAAS reports 62% of students 
grades 6-8 and 68% of students grades 9-12 correctly identify the rate of plate movement as several inches per year. 
College students in our study answer this question at a slightly higher rate of 81 % correct. 

Question 5 is adapted from Clark and Libarkin (2011a) and Clark (201 lb) and colleagues who report students often 
confuse spreading ridges with hot spots and are unable to identify directional motion of plates at boundaries without 
arrows being provided. Although Clark (2011b) does not report frequency of this misconception, our results show that 
only 51% are able to identify a divergent boundary on a diagram without arrows. It is worth noting that our question 
used the words “moving apart” as opposed to “divergent boundary” in order to use more natural student language in 
contrast to scientific vocabulary. Question 6 is adapted from the Geoscience Concept Inventory. Libarkin and 
Anderson (2005a) report 56% of 2,483 students have misconceptions about the location of tectonic plates. These 
results are also similar to our results. 

In the follow-up interviews, participants hold prominent misconceptions in terms of Earth’s plates and how they move. 
In regard to question 5,51% answered the question correctly with the majority of other respondents selecting B (15%) 
or D (33%). One participant stated, “I know volcanoes form when plates move apart however; this diagram doesn’t 
display plates moving apart.” The diagram displays a cross-section illustrating Earth’s crustal plate boundaries with 
oceanic and continental crust. Point A on the diagram represents a divergent boundary, which is where Earth’s plates 
are moving apart. The respondent viewed this as the “ocean being closed off so there is no pressure to build a volcano,” 
which is a commonly held misconception that volcanoes form because of pressure. Another respondent stated, “When 
I hear the word ‘plates’ I think of the ground itself. When plates are moving apart, the mass of ground moves into 
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deeper locations, but not in shallow places like the ocean. So the plates are moving apart at B.” This response supports 
the misconception of not being able to accurately identify a divergent or convergent boundary in addition to what 
happens to plates at the various boundaries. 

Question 6 focuses on the location of Earth’s plates. Fifty one percent of respondents were able to correctly identify 
the location of Earth’s plates in relation to Earth’s surface. However, 36% believe that Earth’s surface and its tectonic 
plates are separate from each other (B). In follow-up interviews, one respondent stated, “The tectonic plates [separate 
from Earth’s surface] are the one’s that move and shift to cause earthquakes. They have to be close to Earth’s surface 
otherwise we wouldn’t feel the earthquakes.” This confirms that students are often confused on the location of Earth’s 
plates. 

EGGS Concept 2: Atoms of different elements combine to make minerals, which combine to make rocks . Rocks 
and minerals are classified by their chemical and physical properties . 


Figure 4. Item Analysis for EGGS Question 2 


02. Which of the following construction and building materials is NOT actually a rock? 

Response Labels 

Item difficulty 

Item discrimination 

a. Slate 

0.17 


b. Brick 

0.70 

0.37 

c. Marble 

0.09 


d. Granite 

0.04 



Figure 5. Item Analysis for EGGS Question 3 


Q3. How deep is an oil well drilled? 

Response Labels 

Item difficulty 

Item discrimination 

a. As deep as a football field is long 

0.31 


b. About a mile 

0.50 

0.42 

c. Halfway to Earth’s center 

0.18 


d. To Earth’s center 

0.01 



Figure 6. Item Analysis for EGGS Question 4 


Q4. We can best determine the environment in which lava or magma solidifies and becomes a rock by using the rock’s 

Response Labels 

Item difficulty 

Item discrimination 

a. Density 

0.23 


b. Mineral grain sizes 

0.06 

0.12 

c. Texture 

0.10 


d. Temperature 

0.18 


e. Chemical composition 

0.43 



Question 2 was adapted from work by Dove (1996) who reports pre-service teachers rarely identify common building 
and construction materials as rocks in interviews. Only seventy percent of students in our study correctly identified 
brick as a non-rock. Question 3 was adapted from Rule (2005) who reported that students in his study lack sufficient 
conceptual framework to accurately predict location of fossil fuels. This misconception was true for 50% of students 
in our study. 

Question 4 was created in response to research by Ault (1982) who found that primary students associate a crumbly 
texture or darker color as being indicative of the oldest rocks and by Ford (2006) who reports that most of the 55 
middle school students studied believe rocks are characterized on the basis of their shape. Nearly one quarter focus 
on superficial physical characteristics when categorizing rocks, like shape. This is similar to Happs’ (1982) report that 
students simply characterize rocks based on their weight. Our results are consistent with this research and show that 
most students mistakenly believe rocks are categorized predominantly by chemical composition or density. 
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In follow-up interviews, participants suggested that oil can be found deeper than a mile, but “it wouldn’t be safe 
because of built up pressure.” Students are confused on the location of fossil fuels and the reasons why fossil fuels 
exist in particular areas. In regards to categorizing rocks, only 6% of students understand that mineral grain sizes are 
used. In follow-up interviews one participant stated, “You are able to identify a rock if you know the chemical 
composition and temperature. For example, I know lava can create crystals and diamonds so from that you are able to 
figure out the temperature and determine where the rock came from.” Another participant also believes temperature 
is used to identify rocks by stating, “Magma is hotter than lava so temperature makes sense.” Rock classification is a 
concept that students consistently struggle with. 

EGGS Concept 3: Earth materials take many different forms as they cycle through the geosphere . Rocks form 
from the cooling of magma, the accumulation and consolidation of sediments, and the alteration of older rocks by 
heat, pressure, and fluids . These three processes form igneous, sedimentary, and metamorphic. 


Figure 7. Item Analysis for EGGS Question 7 


Q7. Primarily, why do sediment grains stick together to form? 

Response Labels 

Item difficulty 

Item discrimination 

a. Pressure 

0.56 

0.48 

b. Heat 

0.17 


c. Erosion 

0.10 


d. Chemical bonding 

0.17 



_ Figure 8. Item Analysis for EGGS Question 8 _ 

Q8. Imagine rock samples were gathered from Earth’s crust in each of the marked locations. Rank the collected rock’s ages in 
order from youngest to oldest. 



Response Labels 

Item difficulty 

Item discrimination 

a. YOUNGEST ^A<B<C<D^ OLDEST 

0.16 


b. YOUNGEST ^D<C<B<A^ OLDEST 

0.23 


c. YOUNGEST ^A<C<B<D^ OLDEST 

0.35 

0.20 

d. YOUNGEST ^B<D<C<A^ OLDEST 

0.10 


e. YOUNGEST ^A<B<D<C^ OLDEST 

0.16 
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_ Figure 9. Item Analysis for EGGS Question 9 

Q9. Compared to Earth’s crust at location A, the crust at location B is 



Response Labels 

Item difficulty 

Item discrimination 

a. Old, cold, and thick 

0.29 


b. Dark, dense, and hot 

0.16 

0.35 

c. Dense, cold, and young 

0.31 


d. Dense, thick, and cold 

0.24 



Question 7 was adapted from research by Stofflett (1993) who reports students do not understand rock formation, and 
often attribute formation to weather conditions across all rock types. Our results show that only half of students identify 
pressure as the primary process creating sedimentary rocks. Questions 8 and 9 were created to measure students’ 
understanding that rocks cycle through the geosphere. Kusnick (2002) found that students interchange common words 
and technical vocabulary carelessly. Two-thirds of students in our study exhibit misconceptions in this domain. 

In follow-up interviews, participants elaborated on why they believe heat causes sedimentary rock formation. One 
participant stated, “Heat melts things. Rocks are formed in the center of the Earth, which is hot, so it molds rocks 
together.” Other participants attributed chemical bonding stating, “Rocks form through different particles such as 
sand, lava, and dirt. They stick together through chemical bonding and form rocks.” 

Question 8 asked students to compare rocks ages based on their location in Earth’s crust. Before answering the 
question, participants tried to make sense of the diagram stating, “I believe the dotted line represents the ocean’s 
current, flow of water, or international lines [North or South].” Two participants believed the rocks found at location 
B are the youngest because North America was discovered last and rocks found at location D are older because Africa 
has been around longer. Other participants attributed evolution in determining the age of rocks stating, “In terms of 
evolution, everything started in Africa [life, rock formation, landforms] so the rocks there would be the oldest.” Lastly, 
some participants credited Pangaea stating, “This drawing represents Pangaea so Africa and North America would 
split first so those rocks would be oldest.” Question 9 asked students to compare Earth’s crust at two different locations 
in terms of age, density, and temperature. A few participants attributed the climate of particular locations stating, “I 
selected cold because the crust at location B looks like Canada and I know it’s cold up there” and “The environment 
in New England is cold so the crust is cold too.” Others focused on density and age stating, “The older something is, 
the more dense it is. With older areas, more dirt piles on top over time to make it more dense.” Lastly, some participants 
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credited soil and density stating, “The soil in New England is not fertile so it’s dense and thick.” These results are 
unique in that these mental conceptions have not been repeated elsewhere. 

EGGS Concept 4: Earth’s rocks allow us to reconstruct Earth’s history, giving both relative and absolute dates . 


Figure 10. Item Analysis for EGGS Question 10 


Q10. Which of the following best describes the relationships between humans (people) and dinosaurs? 

Response Labels 

Item difficulty 

Item discrimination 

a. People and dinosaurs co-existed for about 5,000 years 

0.06 


b. People and dinosaurs co-existed for about 500,000 
years 

0.02 


c. Dinosaurs died out about 5,000 years before people 
appeared on Earth 

0.26 


d. Dinosaurs died out about 5,000,000 years before people 
appeared on Earth 

0.43 


e. Dinosaurs died out about 50,000,000 years before 
people appeared on Earth 

0.23 

0.08 

Figure 11. Item Analysis for EGGS Question 11 

Q11. Which of the figures below do you think most closely represents changes in life on Earth? 



FORMS 


DINOSAURS DISAPPEAR 
HUMANS APPEAR 
DINOSAURS APPEAR 


LIFE APPEARS 


FORMS 


TODAY 



EARTH 



EARTH 

FORMS 



FORMS 


DINOSAURS DISAPPEAR 


LIFE (including dinosaurs 
and humans) APPEARS 


FORMS 


TODAY 



EARTH 


(Figure 11 continued on next page) 
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(Figure 11 continued) 


Response Labels 

Item difficulty 

Item discrimination 

a. 

0.14 


b. 

0.10 


c. 

0.41 


d. 

0.26 

0.32 

e. 

0.09 



_ Figure 12. Item Analysis for EGGS Question 12 _ 

Q12. A geologist in the diagram is standing on marine sedimentary rock containing fossils. What is the order of fossils according 
to their age from oldest to youngest? 



Response Labels 

Item difficulty 

Item discrimination 

a. OLDEST <- trilobite, clam, coral, snail -> YOUNGEST 

0.63 

0.47 

b. OLDEST clam, trilobite, snail, coral -» YOUNGEST 

0.14 


c. OLDEST <- coral, clam, snail, trilobite -> YOUNGEST 

0.07 


d. OLDEST i- clam, trilobite, coral, snail -» YOUNGEST 

0.16 



Question 10 is adapted from the Geoscience Concept Inventory item number 36 (Libarkin & Anderson, 2005a), and 
is based on research from Schoon (1992), Schoon and Boone (1998), Libarkin, Kurdziel, and Anderson (2007), and 
Trend (2001) reporting that many students think humans and dinosaurs coexisted. Our results are consistent with the 
idea that students do not fully understand the large difference in time between the existence of humans and the 
existence of dinosaurs. Only one quarter of students in our study correctly identify that dinosaurs died out more than 
50 million years before the existence of humans. Where are our results markedly differ is that the vast majority of 
students in our study did not say that humans and dinosaurs coexisted; instead, two-thirds of respondents answered 
that dinosaurs died out between 5,000 and 5 million years before humans appeared. These results are consistent with 
Trend (2000) who found students underestimate timescale lengths. We judge this near zero item discrimination to be 
acceptable because of the small number of students that answered this question correctly. One might wonder if 
students are thinking about birds as being a closely linked ancestor of dinosaurs, which is causing some confusion; 
however, none of our interviewees made this claim in describing their thinking. 
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Question 11 is adapted from the Geoscience Concept Inventory item 28 (Libarkin & Anderson, 2005a) and is based 
on research reported by in Libarkin, Kurdziel, and Anderson (2007). Petcovic and Ruhf (2008) report 75% of pre¬ 
service teachers incorrectly answer this question on a pre-test and 55% answer incorrectly on a post-test, suggesting 
that Earth’s timescale is a tenacious misconception. Their data is wholly consistent with our results where 26% answer 
correctly suggesting that students’ misconceptions are pervasive in this domain. 

Question 12 is adapted from research by Dodick and Orion (2003) on their GeoTAT assessment, puzzle 1. Although 
Ault (1982) reports that young children can appropriately employ principles of superposition and original 
horizontality, results from the GeoTAT reveal students give depth as a priority in determining relative age when 
considering horizontal layers. Dodick and Orion (2003) report that about 80% of students can answer this item 
correctly. Our results show a larger percentage of students struggle with this idea as only 63% of students answer 
correctly. 

In follow-up interviews, participants stated that humans and dinosaurs did not coexist; however, they are unsure of 
the time scale of when dinosaurs died and humans appeared. Participants in follow-up interviews believe that when 
Earth formed there was also some form of life stating, “Most likely plant life existed at the time of Earth’s formation.” 
In terms of the principles of superposition and horizontality, participants are aware that fossils found in deeper layers 
of Earth’s crust are older. Taken together, students’ misconceptions about the nature of deep time are widespread. 

EGGS Concept 5: Fossils provide evidence about the types of organisms that lived long ago and the nature of the 
environments at that time . 


Figure 13. Item Analysis for EGGS Question 13 


Q13. Most species of living things alive today 

Response Labels 

Item difficulty 

Item discrimination 

a. Have existed since the time life began. 

0.11 


b. Have existed since the time life began, but new ones 
have now appeared too. 

0.37 


c. Did not exist when life began on Earth. 

0.52 

0.63 


Figure 14. Item Analysis for EGGS Question 14 


Q14. The scientific term “fossil” refers to 

Response Labels 

Item difficulty 

Item discrimination 

a. Remains of once living things. 

0.24 


b. Remains of living things AND traces, like preserved 
footprints. 

0.59 

0.57 

c. Once living things that are now extinct and no longer 
live anywhere on Earth. 

0.17 



Question 13 is adapted from AAAS Project 2061 Science Assessment project, item EN055001, intended to measure 
students’ misunderstanding that all species began at the same time and still exist today (AAAS Project 2061, n.d.). 
AAAS reports 46% of students grade 6-8 and 52% of students grades 9-12 correctly respond that most species living 
today did not exist at the time life began. AAAS reports that 8% of students in their study exhibit this misconception. 
This result is similar to our study where only 11 % of students exhibit this misconception. It does not appear that this 
mistaken notion is as widespread as one might predict. 

Question 14 is heavily modified from AAAS Project 2061 Science Assessment project, item EN013002, to measure 
students’ understanding of how scientists conceive of fossils. AAAS reports 48% of grade 6-8 and 55% of students 
grades 9-12 correctly identify that fossils can be made through replacement or impressions. In our study 59% of 
students answer correctly, whereas 25% believe fossils are remains of once living things. These results are consistent 
with Petcovik and Ruhf (2008), who found students only conceive of fossils as remains of living things. Oversby 
(1996) describes pre-service teachers’ misconceptions that only living things, like plants and animals, that have 
undergone fossilization are considered fossils. In other words, many students would not consider an unfossilized shark 
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tooth or a mold of an animal track a fossil. These results do not speak to research by Dodick and Orion (2003) who 
report students think older fossils are always less complex than more recent fossils. 

In follow-up interviews participants leaned on the theory of evolution when thinking about species that exist today 
stating, “With evolution we know that new life forms have appeared in addition to what’s existed since life began; 
most living things have evolved over time.” When defining the word “fossil” several participants selected the correct 
answer because they remembered learning vocabulary terms [trace fossils] in elementary and middle school. 

EGGS Concept 6: Earthquakes, mountain building, volcanic activity, and ocean floor features occur at plate 
boundaries as the result of plate movement. 


_ Figure 15. Item Analysis for EGGS Question 15 _ 

Q15. The maps below show the position of the Earth’s continents and oceans. The % symbols on the maps mark the locations 
where VOLCANIC ERUPTIONS occur on land. Which map do you think most closely represents the places where these 
volcanoes are typically observed? 











o*;* )/ 


# 8 *;- )/ ^ 





A. Mostly along the margins of the Pacific and Atlantic Oceans B. Mostly along the margins of the Pacific Ocean 





Response Labels 

Item difficulty 

Item discrimination 

a. Mostly along the margins of the Pacific and Atlantic 
Oceans 

0.26 


b. Mostly along the margins of the Pacific Ocean 

0.38 

0.38 

c. Mostly in warm climates 

0.12 


d. Mostly on continents 

0.09 


e. Mostly on islands 

0.15 
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Figure 16. Item Analysis for EGGS Question 16 


Q16. Why are volcanoes and earthquakes often studied together? 

Response Labels 

Item difficulty 

Item discrimination 

a. Earthquakes force lava upwards 

0.03 


b. Earthquakes cause volcanoes 

0.07 


c. Both occur at plate boundaries 

0.49 

0.48 

d. Both are caused by pressure pushing upward from 
underneath 

0.41 



_ Figure 17. Item Analysis for EGGS Question 17 _ 

Q17. Which of the plate boundaries shown below as cross-sections are most likely to produce volcanoes? 



Separating plates under the ocean Colliding plates under continents 


Response Labels 

Item difficulty 

Item discrimination 

a. Separating plates under the ocean 

0.30 

0.23 

b. Colliding plates under continents 

0.70 



Question 15 is adapted from the Geoscience Concept Inventory, item number 13 (Libarkin & Anderson, 2006) and 
measures students’ understanding of volcano locations at plate boundaries. Petcovik and Ruhf (2008) report 55% of 
pre-service teachers answer correctly on a pre-test, increasing to 54% answering correctly on a post-test. In our study 
62% of students incorrectly identify the locations of volcanoes and plate boundaries. This is wholly consistent with 
the misconception that continental margins are plate boundaries. Only 38% of our respondents correctly answered 
choice B. This item appears to be working as anticipated and consistent with research reported by Parham (2010) and 
colleagues. They created the 10-item student-supplied-response Volcanic Concept Survey and administered it to 672 
students across five universities. The goal of the survey was to identify conceptual domains where introductory 
geology students had deficits in their understanding of volcanoes. Their study uncovered four dominant 
misconceptions about the formation of volcanoes: 1) Volcanoes only form near bodies of water (17%); 2) Volcanoes 
are common only in areas near the equator or other warm areas (15.2%); 3) Volcanoes appear in areas of rocky terrain 
(6%); and 4) There is no pattern to geographic locations where volcanoes form (50.4%). 

Question 16 is adapted from research by Barrow and Haskins (1996) who administered a questionnaire to 186 college 
students prior to college level instruction—it was assumed that all students had experienced at least a modicum of 
Earth science instruction during their K-12 learning experiences. When asked why earthquakes and volcanoes are 
often studied together, 17% of students described that earthquakes cause volcanoes, mostly because earthquakes force 
lava upward. Only 6% suggested that volcanoes form when plates collide or occur at plate boundaries. In our study 
about half answered correctly that volcanoes and earthquakes are often studied together because of action at plate 
boundaries. We also found that 41% of students answered that both are caused by pressure pushing upward from 
underneath, suggesting that this item is detecting a widespread misconception. 

Question 17 was created from a surprising lack of published research surrounding sea floor spreading. Experienced 
geology teachers know that students naturally assume that divergent plate boundaries result in deep chasms at the 
bottom of the ocean, rather than submarine mountain ranges caused by upwelling magma. Interview research of 26 
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middle school students by Ford and Taylor (2006) found that some students think there are large gaps or spaces 
between Earth’s plates. Seventy percent of our students believe that volcanoes are more likely to result from colliding 
continental plates than diverging oceanic plates. These results suggest that this is a rich area for future exploratory 
research. 

In follow-up interviews several participants believe volcanoes form when two pieces of land come together. One 
participant selected A (mostly along the margins of the Pacific and Atlantic oceans) stating, “volcanoes form when 
oceanic and continental plates come together.” Another participant selected D (mostly on continents) because “there 
are some volcanoes in water but mostly are underground and on land [not in water].” Another participant stated 
volcanoes form when “two pieces of land come together at an angle so they probably form most often on islands.” 
This participant went on to address continental drift stating, “I know with continental drift the plates didn’t line up. 
One piece of land overtook another piece of land and the core came up through the middle of the Earth. But I know 
that volcanoes make new land.” Another participant specifically stated why volcanoes do not form in warm climates 
stating, “It already hot in warm climates so there wouldn’t be volcanoes.” 

When asking students why volcanoes and earthquakes are studied together in follow-up interviews, most participants 
attributed the linking concept as pressure. One participant stated, “Both are caused by pressure pushing up from 
underneath. The center of the Earth has a lot of energy and movement, which causes pressure. Volcanoes blow up 
[erupt] through a release valve from too much pressure. Earthquakes don’t have a release valve so the plates move 
[instead of erupting]. Earthquakes and volcanoes are the same concept but they occur in different locations.” Other 
participants believe earthquakes force lava upwards stating, “Earthquakes cause disruptions in volcanoes and if it’s 
intense enough then it will cause the volcano to erupt.” 

When asking students which plate boundaries are most likely to produce volcanoes in follow-up interviews several 
participants selected colliding plates under continents. One participant attributed pressure stating, “With the plates 
separating there wouldn’t be any force/pressure going up.” Another participant stated, “When two pieces of land come 
together, they push up to form volcanoes.” Another participant attributed mountains stating, “This diagram looks like 
it’s forming a mountain and I know that mountains are sometimes volcanoes.” These domains represent an important 
area for targeted curriculum development to enhance students’ understanding. 

EGGS Concept 7: The Earth has a layered structure with a dense metallic core, hot convecting mantle, and a 
brittle crust. 


_ Figure 18. Item Analysis for EGGS Question 18 

Q18. The most dense part of Earth is near which point? 



Response Labels 

Item difficulty 

Item discrimination 

a. North pole 

0.07 


b. Earth’s center 

0.67 

0.35 

c. South pole 

0.07 


d. Above South Pole, but below center of Earth 

0.19 
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Figure 19. Item Analysis for EGGS Question 19 


Q19. The very center of Earth is mostly made of 

Response Labels 

Item difficulty 

Item discrimination 

a. Gases. 

0.06 


b. Liquids. 

0.22 


c. Solids. 

0.40 

0.48 

d. A combination of gases, liquids, and solids. 

0.32 



Question 18 was adapted from research by Marques and Thompson (1997) to measure students’ understanding of the 
changing density of Earth’s interior. They found that 42% of students believe the densest part of the Earth is not the 
center, but a position between the center and South Pole. This is consistent with our results showing that nearly 20% 
of students believe the densest part of Earth is south of the equator. Only 67% of our respondents pinpoint Earth’s 
center as the densest point. These results add weight to tangential research by Sneider and Ohadi (1998) that students’ 
misconceptions about gravity, and the nature of which direction is up in space, are well poised to interfere with geology 
instruction. 

Question 19 was included because the literature suggests that a significant number of students are unable to accurately 
identify the physical phase of either the core or Earth’s mantle. King (2000, 2008) reports many students conceptualize 
the Earth as a sphere of molten, liquid magma contained by Earth’s thick, solid crust. Clark (2011) and colleagues 
also report that college students in their study conceive of the mantle as being predominantly liquid. McAllister (2014) 
found that 22% of the 92 undergraduates she interviewed have no coherent conceptual model of Earth’s interior that 
reflect current scientific understanding. In fact only 9% of the 92 undergraduates could describe and sketch 
comprehensive conceptual models that were accurate. These results are similar for international students studied by 
Capps, McAllister, and Boone (2013). In our study we found that only 40% of students believe the very center of the 
Earth is solid and 32% mistakenly believe the very center of the Earth is a combination of gases, liquids, and solids. 
These results confirm that robust misconceptions exist regarding the nature of Earth’s interior. 

In follow-up interviews several participants describe the densest part of the Earth as being at the South Pole. One 
participant attributed weight and gravity stating, “It’s at the bottom because the weight and gravity are pushing down 
which makes it more dense.” Another participant compared the North and South Poles stating, “At the North Pole it’s 
just an ocean so it’s not as dense. The South Pole has Antarctica with lots of ice and land mass which makes it very 
dense.” In follow-up interviews some participants believe the center of the Earth is mostly made of a combination of 
gases, liquids, and solids stating, “I know the gases build pressure. There is heat which needs something to burn which 
creates molten lava and solids.” These results lend weight to the power of these items to successfully identify well- 
known misconceptions. 

EGGS Concept 8: Earth’s interior is heated primarily by radioactive decay and gravitational energy . 


Figure 20. Item Analysis for EGGS Question 20 


Q20. Most scientists think that the inner most parts of Earth are incredibly hot. This high temperature mostly comes from energy 

Response Labels 

Item difficulty 

Item discrimination 

a. Emitted from radioactive decay 

0.26 

0.00 

b. Leftover from the not-yet-cooled Earth from its 
formation 

0.38 


c. From the Sun 

0.13 


d. Due to Earth’s gravity 

0.23 
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Figure 21. Item Analysis for EGGS Question 21 


Q21. If a 5 kilogram radioactive rock has a half-life of 50,000 years, this means that in 50,000 years, the rock will be 

Response Labels 

Item difficulty 

Item discrimination 

a. 1 % kilograms 

0.04 


b. 2 kilograms 

0.49 


c. 5 kilograms 

0.25 

0.00 

d. 10 kilograms 

0.22 



Question 20 is adapted from the Geoscience Concept Inventory , item 27 (Libarkin & Anderson, 2006) who report 
students think Earth’s hot interior is due to insufficient cooling rather than energy from radioactive decay. Petcovik 
and Ruhf (2008) report 25% of pre-service teachers are able to answer this item correctly on a pre-test and 41% answer 
correctly on a post-test. Our results were nearly identical, suggesting this item is well suited to identify this 
misconception, although it does not readily reveal what the underlying physical mechanism students are employing in 
their thinking. It should be noted that we find this near zero item discrimination to be acceptable because of the small 
number of students that answered this question correctly. 

Question 21 is adapted from research by Prather (2005) who reports students have deeply entrenched misconceptions 
about half-life and radioactive decay. His interviews with college students revealed that students often equate 
radioactive decay with the disintegration of matter. In our study, 49% of students have misconceptions related to half- 
life, which is consistent with Prather’s (2005) results. It should be noted that we find this near zero item discrimination 
to be acceptable because of the small number of number students that answered this question correctly. 

In follow-up interviews several participants attribute heat leftover from the not-yet-cooled Earth from its formation to 
the heat in the center of the Earth. One participant admitted, “I’m not sure what radioactive decay is but I know when 
Earth formed it was hit by something else and knocked into orbit. It was really hot at first then cooled a little to allow 
plant life to form and then we had the great freeze. So I’m pretty sure Earth is still cooling.” Another participant 
attributed gravity to the hot inner parts of the Earth stating, “gravity causes heat.” In follow-up interviews some 
participants were unable to calculate the half-life of a radioactive rock. Some participants believed the rock wouldn’t 
decrease in weight because, “The rock is already formed so it can’t change over time. It could combine with other 
particles so it could weigh more but not less.” 
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EGGS Concept 9: Rocks are chemically and physically weathered into smaller pieces, which are transported 
(eroded) by gravity, water, ice, and wind. 


_Figure 22. Item Analysis for EGGS Question 22_ 

Q22. In the cross-section of a river valley shown below, there are loose rocks on top of a solid rock layer. Can water make the 
valley deeper? 



Response Labels 

Item difficulty 

Item discrimination 

a. No, valleys do not get deeper. 

0.02 


b. No, valleys can get deeper, but water cannot make them 
deeper. 

0.16 


c. Yes, water can make the valley deeper by breaking off 
pieces of rock from the solid rock layer of the valley 
and by carrying loose rocks away from the valley. 

0.64 

0.37 

d. Yes, water can make the valley deeper by carrying 
loose rocks away from the valley, but water cannot 
break off pieces of rock from the solid rock layer of the 
valley. 

0.18 



Figure 23. Item Analysis for EGGS Question 23 


Q23. Can water dissolve rock material and move the dissolved material to a new location, and deposit the dissolved rock material 
as solid rock? 

Response Labels 

Item difficulty 

Item discrimination 

a. Yes, water can dissolve rock, move dissolved solid 
rock material to a new location, and deposit dissolved 
rock material as solid rock. 

0.47 

0.33 

b. No, although water can dissolve rocks, water cannot 
move dissolved rock material to a new location. 

0.08 


c. No, although water can dissolve rock and can move 
dissolved rock material to a new location, dissolved 
material cannot become solid rock. 

0.30 


d. No, water cannot dissolve rock. 

0.15 



To measure student’s understanding of weathering and erosion we adapted two existing items from the AAAS Project 
2061 Science Assessment effort, WEO15003 and WE032003. The first item was designed to test for the common 
misconception that landforms look similar today as they did many millions of years ago. For example, a river on earth 
today hasn’t changed over time (Dove, 1998; Trend, 1998) and water cannot make a valley deeper or break rock 
(AAAS Project 2061, n.d.). AAAS reports that 60% of students grades 6-8 and 58% of 9-12 correctly answer C, with 
remaining 40% evenly distributed across other answers. The second item was designed to test the misconception that 
water cannot deposit dissolved rock as solid rock and cannot dissolve rocks (AAAS Project 2061, n.d.). AAAS reports 
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that only 35% of students grades 6-8 and 39% of 9-12 correctly answer A, with 34% answering C. These results are 
similar to our results. 

In combination, these two items appear to be working as designed. However, these items are not reflective of research 
by Dove (1997) reporting geography students being unable to distinguish between weathering and erosion or research 
by Blake (2005) reporting students often use unscientific descriptions or Martinez, Bannan, and Kitsantas (2012) 
reporting students appealing to unnatural explanations of changes to Earth’s surface. Erosion and weathering appear 
to be a relatively unexplored conceptual domain that could be rich and fruitful for future research studies. 

In follow-up interviews several participants believe that water can make the valley deeper but is unable to deposit 
larger rocks in other places. One participant stated, “Loose rocks makes me think of sediments but I don’t think solid 
rocks can get carried away because they are too big and heavy. Water can erode rock over time but it is unable to carry 
away rocks.” Other participants believe valleys are able to get deeper but water cannot make them deeper. One 
participant stated, “Water can cause erosion but it can’t break rock. Breaking off rock would make the valley more 
shallow and it will eventually be completely gone [the sides of the valley will no longer be there].” This confirms that 
students still struggle with differentiating between weathering, erosion, and deposition. 

Regarding the second item in follow-up interviews most participants believe water is able to dissolve rock and move 
it to a new location. However, some participants believe the dissolved material cannot become solid rock. Other 
participants believe that water cannot dissolve rock stating, “Water can’t dissolve rock but it can move it.” 

EGGS Concept 10: Soil is formed by weathered rocks and decayed organic materials . 


Figure 24. Item Analysis for EGGS Question 24 


Q24. About how old is most of the soil near the location where you live? 

Response Labels 

Item difficulty 

Item discrimination 

a. Tens to 100s of years old 

0.25 

0.02 

b. Thousands of years old 

0.26 


c. Millions of years old 

0.27 


d. As old as Earth—it formed at about the same time 

0.22 



Figure 25. Item Analysis for EGGS Question 25 


Q25. About how deep is the soil where you live? 

Response Labels 

Item difficulty 

Item discrimination 

a. A few inches, or a maybe a few feet, deep 

0.26 

0.22 

b. As deep as a football field is long or more 

0.33 


c. Several miles deep 

0.32 


d. To nearly the center of Earth 

0.09 
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_ Figure 26. Item Analysis for EGGS Question 26 _ 

Q26. The figure below shows three funnels holding 10 kilograms of different sized rocks. In which case will water poured into 
the top pass through it MOST QUICKLY ? 


Response Labels 

Item difficulty 

Item discrimination 

a. 

0.58 

0.48 

b. 

0.08 


c. 

0.34 



These three items were developed based on limited research extant in scholarly literature base related to soils. Happs 
(1984) reports that 53% students he studied believe soil was formed as the point of Earth’s formation, and 33% thought 
the average depth of soil is about one meter deep, and 23% believes soil does not evolve. Similarly, Hayhoe’s (2013) 
review of soil science education curriculum materials resulted in his conclusion that students lack a consistent 
conceptual framework related to soil science. No empirical research seems to exist regarding the concept of porosity, 
even though experienced teachers tacitly know students struggle with this idea. 

When asking students to elaborate in follow-up interviews regarding their answer choices for the age of soil, it is 
apparent that the majority of participants were guessing. This is consistent with the quantitative results. One participant 
stated, “I’m not really sure of how soil is formed; maybe it’s slow decomposition of organic material over time. I 
guessed C through process of elimination. A and B are too short of a time frame and soil wasn’t here when Earth 
formed.” Others simply stated, “I had no idea, so I guessed.” Several students also admitted to guessing when asked 
about the depth of soil. Others discussed their experience with planting trees but weren’t sure of the exact depth stating, 
“I think it’s a few inches or maybe a few feet but it could also be as deep as a football field is long; but no deeper than 
that.” Another participant believed soil to be several miles deep stating, “I thought about how deep the crust is and 
soil is part of the crust so it has to be several miles deep.” The last item addresses porosity and the majority of students 
understand this concept. However, a few students selected B and in follow-up interviews explained their reasoning. 
One student stated, “The rocks in funnel A wouldn’t pass through as quickly because the rocks are huge and they 
would absorb some of the water. The rocks in funnel C are small so the water would capture some of the rocks and 
pass through but not as quickly as the rocks in funnel B.” The three items related to soils appear to confirm these 
above results, and many students appear to simply be posing their best guess. 
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EGGS Concept 11: Landforms result from the interplay between processes that create crust (plate movement, crust 
uplift, and sedimentary rock formation) and those that destroy crust (weathering and erosion). These interactions 
occur at a variety of time scales. 


Figure 27. Item Analysis for EGGS Question 27 


Q27. How old are the Rocky Mountains? 

Response Labels 

Item difficulty 

Item discrimination 

a. As old as Earth (formed about the same time) 

0.12 


b. As old as most mountain chains on Earth (except the 
newest volcanoes) 

0.73 


c. As old as dinosaurs 

0.12 

-0.09 

d. As old as the United States Constitution 

0.03 



Figure 28. Item Analysis for EGGS Question 28 


Q28. In general, which direction do rivers flow? 

Response Labels 

Item difficulty 

Item discrimination 

a. Downhill 

0.40 

0.48 

b. North to south 

0.22 


c. West to east 

0.04 


d. Both B & C 

0.34 



There is surprisingly little research on student’s conceptual understanding of landforms. Francek’s (2013) report on 
500 misconceptions lists only karst topography when discussing landforms. Question 27 was devised to measure 
students’ thinking about the variety of timescales encountered when learning geology. Our results show that 73% of 
students hold the misconception that all mountains are the same age. It should be noted that we find this near zero 
item discrimination to be acceptable because of the small number of students that answered this question correctly. 
Question 28 is adapted from research by Trend, Everett, and Dove (2000) who reported students believe rivers always 
flow north to south or west to east rather than downhill. Our results are consistent with Trend and colleagues (2000), 
with 60% of students suggesting rivers flow north to south and/or west to east. 

When asking students to elaborate on their understanding of mountain ages, many students believe all mountains are 
around the same age but it appears students choose this answer by a process of elimination. One participant stated, “I 
know that all geologic features change over a long period of time so the answer couldn’t be A (as old as Earth). I’m 
not sure how old dinosaurs are and the US constitution is too young. So the logical answer is B.” In follow-up 
interviews, many students believe rivers flows north to south. Some participants utilized rivers close to their home 
stating, “In California, the Sacramento river flows north to south so I assume all rivers probably flow in that same 
direction. Other participants believe rivers flow west to east. One student accredited the stars stating, “The sun has no 
correlation to the direction but the stars rise in the east and set in the south [sic] so rivers probably run west to east.” 
These two items appear to be working as designed. 

CONCLUSION 

In the service of establishing the validity and reliability of a tool to efficiently determine students’ understanding of a 
broad consensus of important geology concepts, we administered the final version of the EGGS Exam of GeoloGy 
Standards to non-science majoring undergraduate students enrolled in introductory college science survey courses, 
and completed follow-up think-aloud interviews. In answer to a motivating question for this study, which geology 
misconceptions do introductory college geology students harbor?, the EGGS is successfully able to identify 
misconceptions described in much of scholarly literature surrounding geoscience education, in addition to identifying 
several new misconceptions. 

Overall, 172 participants scored an average of 42% correct with a standard deviation of 0.014. The Cronbach’s alpha 
was 0.69 and the average item discrimination was 0.34. Taken together with supporting interview data, we judge that 
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the EGGS is a valid and reliable investment to efficiently elicit students’ misconceptions. Appendix A provides a 
synthesized list of misconceptions aligned with the 11 most common conceptual ideas identified from an overlap of 
the national standards documents. Taken together, the results of this effort suggest that EGGS is now ready for wide- 
scale deployment in the geoscience education community, as one of a growing number of tools available to help 
faculty improve teaching and learning. 
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APPENDIX 

Synthesized List of Misconceptions Aligned with EGGS Conceptual Targets 


Consensus Concept 

Misconceptions (Frequency) 

1. Earth’s crust is broken into plates, which 
slowly move in relationship to each other, 
driven by convection currents in the mantle. 

Earth’s plates move apart at oceanic-continental convergent boundaries (15%) 

Tectonic plates are separate from Earth’s surface (36%) 

2. Atoms of different elements combine to 
make minerals, which combine to make rocks. 
Rocks and minerals are classified by on their 
chemical and physical properties. 

Rocks are classified by density, texture, temperature, or chemical composition 
(94%) 

The construction and building materials, slate, marble, and granite, are not 
rocks (30%). 

An oil well is drilled halfway to Earth’s center (18%); As deep as a football 
field is long (31%). 

3. Earth materials take many different forms 
as they cycle through the geosphere. Rocks 
form from the cooling of magma, the 
accumulation and consolidation of sediments, 
and the alteration of older rocks by heat, 
pressure, and fluids. These three processes 
form igneous, sedimentary, and metamorphic. 

Primarily, heat (17%) or chemical bonding (17%) causes sediment grains to 
stick together to form sedimentary rocks. 

Rocks form in the center of the Earth (not present in quantitative results; 17% 
of participants interviewed). 

Earth’s crust found near mid-oceanic ridges is generally older than crust found 
near North America (31%) 

North America was discovered last; therefore, the rocks in North American are 
younger than the rocks in Africa (not present in quantitative results; 50% of 
participants interviewed) 

4. Earth’s rocks allow us to reconstruct 
Earth’s history, giving both relative and 
absolute dates. 

People and dinosaurs co-existed (8%). 

Life appeared when Earth formed (14%). 

Dinosaurs died out about 5,000 years before peopled appeared on Earth (26%). 

5. Fossils provide evidence about the types of 
organisms that lived long ago and the nature 
of the environments at that time. 

Most species of living things alive today have existed since the time life began, 
but new ones have appeared too (37%). 

6. Earthquakes, mountain building, volcanic 
activity, and ocean floor features occur at 
plate boundaries as the result of plate 
movement. 

Volcanoes are typically observed along the margins of the Pacific and Atlantic 
Oceans (26%), mostly on continents (9%), or mostly on islands (15%). 

Volcanoes and earthquakes are studied together because both are caused by 
pressure pushing up from underneath (41%). 

Colliding plates under continents are most likely to produce volcanoes (70%). 

7. The Earth has a layered structure with a 
dense metallic core, hot convecting mantle, 
and a brittle crust. 

The most dense part of Earth is near the South pole (7%) or above the South 
pole but below the center of Earth (19%). 

The very center of the Earth is mostly made of a combination of gases, liquids, 
and solids (32%). 

8. Earth’s interior is heated primarily by 
radioactive decay and gravitational energy. 

The innermost parts of the Earth are incredibly hot. This high temperature 
mostly comes from energy leftover from the not-yet-cooled Earth from its 
formation (38%). 

Unable to calculate half-life (75%). 
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Synthesized List of Misconceptions Aligned with EGGS Conceptual Targets (continued) 


Water can make a valley deeper by carrying loose rocks away from the valley, 
but water cannot break off pieces of rock form the solid rock layer of a valley 
(18%). 

Valleys can get deeper, but water cannot make them deeper (16%). 

Water cannot dissolve rock (15%). 

Although water can dissolve rock and can move dissolved rock material to a 
new location, dissolved material cannot become solid rock (30%). 

10. Soil is formed by weathered rocks and Soil is as old as Earth—it formed about the same time (22%). 
decayed organic materials. 

The depth of soil is nearly to the center of Earth (9%). 

11. Landforms result from the interplay Rivers flow north to south (22%). 

between processes that create crust (plate 

movement, crust uplift, and sedimentary rock Rivers flow north to south and west to east (34%). 

formation) and those that destroy crust 
(weathering and erosion). These interactions 
occur at a variety of time scales. 


9. Rocks are chemically and physically 
weathered into smaller pieces, which are 
transported (eroded) by gravity, water, ice, 
and wind. 
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Science Survey 
EGGS161205 

This is a voluntary survey to help us improve and focus this class. Your answers nor your participation 
impacts your grade. 

Always select the BEST answer. 


1. About how far do Earth’s continents—and plates they ride on—move in a single year? 

a. A few inches 

b. A few hundred feet 

c. A few miles 

d. Continents and plates do not really move 

2. Which of the following construction and building materials is NOT actually a rock? 

a. Slate 

b. Brick 

c. Marble 

d. Granite 

3. How deep is an oil well drilled? 

a. As deep as a football field is long 

b. About a mile 

c. Halfway to Earth’s center 

d. To Earth’s center 

4. We can best determine the environment in which lava or magma solidifies and becomes a rock by using the rock’s 

a. density. 

b. mineral grain sizes. 

c. texture. 

d. temperature. 

e. chemical composition. 
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Cross-section illustrating Earth's crustal plate boundaries 
showing oceanic crust and continental crust (not to scale) 



5. Which letter on the diagram above illustrates a location where Earth’s plates are moving APART ? 

a. A 

b. B 

c. C 

d. Both locations A & B are moving apart 


6. Scientists often talk about Earth’s tectonic plates and their role in mountain formation, volcanoes, and 
earthquakes. Which of the following figures most closely represents the location of Earth’s tectonic plates? 




Item #6 is adapted from GCIv.3, Libarkin, J. C., & Anderson, S. W. (2005). Assessment of learning in entry-level 
geoscience courses: Results from the Geoscience Concept Inventory. Journal of Geoscience Education , 53(4), 394-401, 
funded by NSF 0127765, 0350395, 0717790, 0717589 
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7. Primarily, why do sediment grains stick together to form sedimentary rocks? 

a. Pressure 

b. Heat 

c. Erosion 

d. Chemical bonding 


Use the drawing below to answer the next two questions. 



8. Imagine rock samples were gathered from Earth’s crust in each of the marked locations. Rank the collected rock’s 


ages in order from youngest to oldest. 


a. 

Youngest <- A<B<C<D 

-> Oldest 

b. 

Youngest <- D<C<B<A 

-> Oldest 

c. 

Youngest <- A<C<B<D 

-> Oldest 

d. 

Youngest <- B<D<C<A 

-> Oldest 

e. 

Youngest <- A<B<D<C 

-> Oldest 

Compared to Earth’s crust at location A, the crust at location B is 

a. 

old, cold and thick. 


b. 

dark, dense, and hot. 


c. 

dense, cold, and young. 


d. 

dense, thick, and cold. 
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10. Which of the following best describes the relationships between humans (people) and dinosaurs? 

a. People and dinosaurs co-existed for about 5,000 years 

b. People and dinosaurs co-existed for about 500,000 years 

c. Dinosaurs died out about 5,000 years before people appeared on Earth 

d. Dinosaurs died out about 5,000,000 years before people appeared on Earth 

e. Dinosaurs died out about 50,000,000 years before people appeared on Earth 


11. Which of the figures below do you think most closely represents changes in life on Earth over time? 


TODAY 



HUMANS APPEAR 
DINOSAURS DISAPPEAR 
DINOSAURS APPEAR 


TODAY 


EARTH 

FORMS 


-LIFE APPEARS 


TODAY 


B 


-DINOSAURS DISAPPEAR 
-HUMANS APPEAR 
-DINOSAURS APPEAR 


TODAY 


- LIFE APPEARS 


-HUMANS APPEAR 
-DINOSAURS DISAPPEAR 


-DINOSAURS APPEAR 


-LIFE APPEARS 


EARTH 

FORMS 


EARTH 

FORMS 


TODAY 



HUMANS APPEAR 



-DINOSAURS DISAPPEAR 



-DINOSAURS APPEAR 


D 


1 

1 

LU i 

I 

l 


-LIFE APPEARS 



-DINOSAURS DISAPPEAR 


-LIFE (including dinosaurs 
and humans) APPEARS 


EARTH 

FORMS 


EARTH 

FORMS 


Some items are adapted from GCIv.3, Libarkin, J. C., & Anderson, S. W. (2005). Assessment of learning in entry-level geoscience courses: 
Results from the Geoscience Concept Inventory. Journal of Geoscience Education, 55(4), 394-401, funded by NSF 0127765,0350395, 0717790, 
0717589. 
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Item adapted from Dodick, J., & Orion, N. (2003). Measuring student understanding of geological time. Science 
Education, #7(5), 708-731. 


12. A geologist in the diagram is standing on marine sedimentary rock containing fossils. What is the order of fossils 
according to their age from oldest to youngest? 

a. OLDEST <- trilobite, clam, coral, snail -> YOUNGEST 

b. OLDEST <- clam, trilobite, snail, coral -» YOUNGEST 

c. OLDEST <- coral, clam, snail, trilobite -> YOUNGEST 

d. OLDEST <- clam, trilobite, coral, snail -» YOUNGEST 


13. Most species of living things alive today 

a. have existed since the time life began. 

b. have existed since the time life began, but new ones have now appeared too. 

c. did not exist when life began on Earth. 

14. The scientific term “fossil” refers to 

a. remains of once living things. 

b. remains of living things AND traces, like preserved footprints. 

c. once living things that are now extinct and no longer live anywhere on Earth. 
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15. The maps below show the position of the Earth’s continents and oceans. The % symbols on the maps mark the 
locations where VOLCANIC ERUPTIONS occur on land. 

Which map do you think most closely represents the places where these volcanoes are typically observed? 



A. Mostly along the margins of the Pacific 
and Atlantic Oceans 


B. Mostly along the margins of the Pacific 
Ocean 





Item is adapted from GCIv.3, Libarkin, J. C., & Anderson, S. W. (2005). Assessment of learning in ntry-level geoscience courses: Results from 
the Geoscience Concept Inventory. Journal of Geoscience Education, 55(4), 394-401, funded by NSF 0127765, 0350395, 0717790, 0717589 


16. Why are volcanoes and earthquakes often studied together? 

a. Earthquakes force lava upwards 

b. Earthquakes cause volcanoes 

c. Both occur at plate boundaries 

d. Both are caused by pressure pushing upward from underneath 
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17. Which of the plate boundaries shown below as cross-sections are most likely to produce volcanoes? 



Separating plates under the ocean Colliding plates under continents 



18. The most dense part of Earth is near the point 

a. North Pole 

b. Earth’s center 

c. South Pole 

d. above South Pole, but below center of Earth 


19. The very center of Earth is mostly made of 

a. gases. 

b. liquids. 

c. solids. 

d. a combination of gases, liquids, and solids. 

20. Most scientists think that the inner most parts of Earth are incredibly hot. This high temperature mostly comes 
from energy 

a. emitted from radioactive decay. 

b. leftover from the not-yet-cooled Earth from its formation. 

c. from the Sun. 

d. due to Earth’s gravity. 

21. If a 5 kilogram radioactive rock has a half-life of 50,000 years, this means that in 50,000 years, the rock will be 

a. 1 % kilogram. 

b. 2Vi kilograms. 

c. 5 kilograms. 

d. 10 kilograms. 
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22. In the cross-section of a river valley shown below, there are loose rocks on top of a solid rock layer. Can water 
make the valley deeper? 

a. No, valleys do not get deeper. 

b. No, valleys can get deeper, but water cannot make them deeper. 

c. Yes, water can make the valley deeper by breaking off pieces of rock from the solid rock layer of the valley 
and by carrying loose rocks away from the valley. 

d. Yes, water can make the valley deeper by carrying loose rocks away from the valley, but water cannot break 
off pieces of rock from the solid rock layer of the valley. 



Items #22 & 23 are adapted from AAAS Project 2061 Science Assessment Website (n.d.) at http://assessment.aaas.org/ 


23. Can water dissolve rock material and move the dissolved material to a new location, and deposit the dissolved 
rock material as solid rock? 

a. Yes, water can dissolve rock, move dissolved solid rock material to a new location, and deposit dissolved 
rock material as solid rock. 

b. No, although water can dissolve rocks, water cannot move dissolved rock material to a new location. 

c. No, although water can dissolve rock and can move dissolved rock material to a new location, dissolved 
material cannot become solid rock. 

d. No, water cannot dissolve rock. 

24. About how old is most of the soil near the location where you live? 

a. Tens to 100s of years old 

b. Thousands of years old 

c. Millions of years old 

d. As old as Earth—it formed at about the same time 

25. About how deep is the soil where you live? 

a. A few inches, or a maybe a few feet, deep 

b. As deep as a football field is long or more 

c. Several miles deep 

d. To nearly the center of Earth 
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26. The figure below shows three funnels holding 10 kilograms of different sized rocks. In which case will water 
poured into the top pass through it MOST QUICKLY ? 



27. How old are the Rocky Mountains? 

a. As old as Earth (formed about the same time) 

b. As old as most mountain chains on Earth (except the newest volcanoes) 

c. As old as dinosaurs 

d. As old as the United States Constitution 

28. In general, which direction do rivers flow? 

a. Downhill 

b. North to south 

c. West to east 

d. Both B & C above 
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NOTES 
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